Method for processing an endoscopy image

ABSTRACT

In a method for processing a generated digital endoscopy image of a patient, those pixels that depict a body region of a patient are determined as image points in the endoscopy image, in a processor. In the processor, a color value is derived for image point using the value of the associated pixel, and an area portion associated with each image point is derived for the color value. A color area is derived for each color value, as the sum of all area portions of the image points having that color value. An evaluation measure of the endoscopy image is then implemented in the processor using the respective color areas.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention concerns a method to process a digital endoscopy image ofa patient that is generated.

2. Description of the Prior Art

The endoscopic examination or treatment of patients is a widespreadmedical measure. In endoscopic interventions, endoscopy images (normallydigital) of the patient are generated. The images are displayed on amonitor to the endoscopist. An endoscopy is frequently performed in thegastrointestinal tract of a patient, for example. The endoscopistconducts a visual observation and assessment of the images by observing,for example, color distribution and texture of the mucous membrane orsurface of the gastrointestinal tract that is imaged in the endoscopyimage. From this, the endoscopist obtains findings and combines thesewith his personal experiences and optional additional information aboutthe examined patient. The quality of this procedure therefore decisivelydepends on the qualification of the endoscopist.

In various patients, follow-up examinations—i.e. additional endoscopiesat later points in time—are always conducted again in order to track theprogress of an illness in the patient, for example. In the knownprocedure, the respective current examination result—i.e. the currentlyacquired endoscopy images—can be only compared qualitatively withearlier examinations. For example, for this purpose the endoscopist candraw upon archived endoscopy images or proceed according to his ownmemories.

The goal of an endoscopy is, for example, to detect or to diagnose aninflammation in the gastrointestinal tract of a patient. The endoscopistmakes a judgment as to whether the respective examined patient hasinflammations or not purely via the observation of the endoscopyimage—i.e. his personal color perception—using the shown patientsurface. The endoscopist is hereby reliant upon his subjectiveperception, as well as on the color differences that are perceptible inthe first place to the human eye.

The visual perception of humans takes place via receptors that arelocated on the retina. These receptors are rods for the light/darkcontrast, cones for color perception. Cones are present in three formsthat have their sensitivity maximum in the spectral ranges “red”,“green” and “blue”. Colors can therefore be represented as athree-dimensional property due to the three types of color receptors inhumans. Each combination of excitations of the three cone types by lightradiation that strikes the retina produces a specific color impression.Black (no excitation at all), neutral grey and white (i.e. full andsimultaneous excitation of all three cone types) are thus also likewisecolors that are classified as achromatic colors.

The one-dimensional representation of the spectral colors as it occursvia refraction into rainbows or after a prism (represented as a colorwheel of the chromatic colors) includes only a few color perceptions.Visible radiation is an electromagnetic radiation in the wavelengthrange from 380 nm to 780 nm. A color perceptible to humans thus can bedefined by three parameters. If the absolute brightness is noweliminated, two parameters remain, meaning that all colors can bedepicted in a 2D space together with a one-dimensional luminosity.

The aforementioned color perception is a subjective process thatproceeds differently for every person. Since human color perception or,respectively, image perception of the endoscopy image is decisive forthe endoscopy that is described above, evidence-based medicine is notconducted here. The latter requires quantitative measurement valuesduring the course of an illness, wherein the measurement values must bedeterminable independent of the examiner. Such a thing is also desirablefor endoscopy.

A computer-assisted diagnosis method (CAD) is known from US 2007/0135715A1. A number of endoscopy images which are generated by an endoscopycapsule are assessed there automatically for hemorrhage locations in thepatient.

A method for analytical detection of a macular degeneration is knownfrom AT 503 741 A2. Images of the background of the eye of a patientwhich are generated via fluorescence angiography and similar imagingmethods are thereby classified via a four-stage method.

SUMMARY OF THE INVENTION

It is the object of the present invention to specify an assistive methodthat allows an improved endoscopy.

In other words, the object is to specify a technique or a method withwhich quantitative examination results can be generated in endoscopy.The invention is based on the idea to introduce for this purpose amethod to process a digital image of a patient that is generated.

The object is achieved by a method according to the invention that isbased on a digital endoscopy image of the patient that was previouslygenerated, which digital endoscopy image is processed according to themethod. In the endoscopy image various pixels are determined as imagepoints, wherein the image points are additionally processed later in themethod. At least a portion of those pixels that depict a region of thebody of the patient are selected as image points. In other words, onlyimage contents of the endoscopy image which actually represent an imageof the patient are thus processed in the method. Pixels that (forexample) depict an instrument situated in the field of view of theendoscope are not taken into account.

A color value is now determined for each image point. This occurs usingthe value of the pixel associated with this image point. Moreover, anarea proportion associated with the image point is determined for eachimage point. As is explained further below, the area proportion canhereby relate to the area of the endoscopy image or the real area of thepatient or, secondly, of the portion of the patient that is depicted inthe image.

If all desired or all available image points in the endoscopy image aredefined and their corresponding color values and area proportions aredetermined, a color area is subsequently determined for each determinedcolor value: the sum of all area proportions whose image points have theappertaining color value is hereby calculated. In other words, for eachcolor rotation determined in the endoscopy image in the form of aspecific color value, it is determined what area proportion of theendoscopy image or of the patient has this one color value. Thecorresponding measure of this is the respective color area associatedwith the color value.

Finally, the evaluation measure for the endoscopy image is determinedusing the determined color areas. Again, different variants describedfurther below are also provided for this purpose.

In other words, according to the invention a form of color map in whichthe respective area proportions in the image or on the patient whichhave a defined color or, respectively, are associated with a color groupare determined is thus created for an endoscopy image of the patientthat is acquired. The evaluation measure is determined from thecorresponding color areas according to a specific algorithm or,respectively, specific rules or a specific method. Each endoscopy imageis therefore evaluated in a reproducible and quantitative manner anddelivers a measurement value that enables evidence-based medicine withinthe scope of endoscopy. Subjective color assessments by the endoscopistare eliminated. The evaluation measure is an objectively reproducible,quantitative measurement.

Since the method operates on the pixel values of the digital endoscopyimage, the evaluation measure simply does not run up against theaforementioned limits of the human color perception; rather, it is onlydependent on the technical differentiation capability of the generationof the endoscopy image or, respectively, on the corresponding bitresolution in the generation of the endoscopy image. Within the scope ofthe method, alternative endoscopy techniques can also be used whichdeliver information in the endoscopy image that goes beyond theresolution capability of the human eye but can be used by the method.Diagnoses can therefore also be improved beyond the characteristics ofhuman color perception if information in the endoscopy image that is notperceptible to the human eye enters into the evaluation measure.

As was mentioned above, multiple variants exist for the determination ofthe area proportion.

In a first embodiment of the method, a value correlated with the area ofthe pixel in the endoscopy image that is associated with this imagepoint is determined as an area proportion of said image point. Thedetermined area proportions are thus oriented towards the image area ofthe endoscopy image. Such a determination of an area proportion isparticularly simple. For example, the actual area of a pixel in an imageis measured as an area proportion.

In an alternative embodiment, a value correlated with the area of thepatient that is depicted at the image point is determined as an areaproportion of said image point. Here the area proportion is orientedtowards the actual surface of the body of the patient which is depictedin the image. For example, here the viewing angle of the endoscopetowards the patient surface is to be taken into account in order todetermine the actual patient area that is depicted at the image point.In this method variant, the area proportion is better related to theactual patient and delivers a measurement value which is orientedtowards the actual patient surface.

The area proportion is always a value that is correlated only with thearea, thus does not need to represent the absolute assessed area;rather, it can merely be in relation to this, for example. Givenorientation towards the image area for an image point, the areaproportion can always be counted as “1”, for example.

As was mentioned above, variants also exist for the determination of theevaluation measure:

In a first embodiment, a total area entering into the evaluation measureis determined as a reference with regard to the respective color areas.The color areas can thus be related to the reference and a percentileor, respectively, ratio value can enter into the determination of theevaluation measure. Various possibilities which normally are used in themethod together with the aforementioned possibilities for thedetermination of the area proportions result again for the total area.

In a first embodiment, the image area of the endoscopy image isdetermined as a total area. In combination with the aforementioneddetermination of the area proportions using the areas of the pixels inthe endoscopy image, total area and area proportions are thus orientedtowards the area of the endoscopy image. Alternatively, the sum of allarea proportions can also be determined as a total area if not all imagecontents depict the patient, for example.

In an alternative embodiment, the area of the patient that is depictedin the endoscopy image is determined as a total area. This variant isnormally combined with the aforementioned variant in which the areaproportions are also oriented towards the actual patient surface.

In a preferred embodiment of the method, the evaluation measure isdetermined as a quotient of color area and total area of one, multipleor all color areas. In other words, the evaluation measure then yieldsconclusions about with which proportions specific colors of the patientsurface are present in the endoscopy image.

In another embodiment, a histogram of the color areas across the colorvalues is determined as an evaluation measure. Here the color areas arethus not linked with a total area; rather, only their respectiveproportions are presented in the form of a histogram.

Various possibilities also exist for the determination of the colorvalue of an image point:

For example, the corresponding pixel value of the associated pixel inthe endoscopy image can be selected as a color value.

In a further embodiment, however, the color value is selected so that itis correlated with a pathological property of the patient. For example,only two color values are assigned for the assessment of an endoscopyimage, namely one that corresponds to an inflamed body region of apatient and one that corresponds to a body region of a patient that isnot inflamed. Only one of the two color values is respectivelyassociated with the image points. For this the pixel values of thecorresponding image pixels are evaluated and classified in thecorresponding color values. Each color value therefore corresponds to anentire value range of pixel values of the image pixels.

In a preferred embodiment of the method, value ranges of pixel valuesare therefore used as color values.

Such color values can be understood as color clusters, wherein theproperties of the respective cluster can be selected.

In a particularly preferred variant of this embodiment, the valuesranges are selected using a standard color system. For example, finitelylarge color ranges—thus color ranges of pixel values—are connected toclusters according to the RAL color system (RAL Deutsches Institut furGutesicherung and Kennzeichnung e.V.); for example, all colors betweenRAL-3014 (antique pink) and RAL-3033 (pearl pink) are combined into acluster which then corresponds to a body tissue of the patient that isnot inflamed.

Value ranges—thus the association or, respectively, classification ofwhich pixel values are associated with which color values—can bepredeterminable as a standard from a central location or, respectively,can be loaded into the respective endoscopy system, for example.Endoscopy images that are processed with such a standard association arethen comparable among one another. For this purpose it is merelynecessary that the endoscopy images have comparable color reproduction,thus are color-calibrated, for example. The values ranges can be loadedfrom a standard medium such as a diskette, for example. It is alsopossible to load the standard values from the Internet, thus for examplefrom a central location such as an Internet server.

The pixel values that establish the color of a pixel are normally valuegroups made up of multiple color channels. For example, a pixel thuscomprises a red value, green value and blue value, respectively in theform of a digital value, wherein each value corresponds to a colorchannel. In a preferred embodiment of the method, the color value isdetermined using a mask applied to the color channels. Theclassification of pixels into various color values then takes place viaa mask comparison and is to be implemented particularly simply andquickly. As mentioned above, the method opens up the possibility to alsoprovide the pixels of the endoscopy images with more color channels thanthe human eye could resolve. Classifications—and thus the creation ofthe evaluation measure—can thus exceed the capability of solely anoptical consideration by a human observer.

In an alternative method variant, a luminosity-normalized 2D value isselected as a color value. The decomposition of colors into a 2D colorvalue and a luminosity was explained above. By eliminating theluminosity, the classification then actually takes place only accordingto the color tone and not according to its saturation in the form of theluminosity.

In one advantageous embodiment of the method, image points arerespectively determined as groups of pixels in the endoscopy image. Inother words, the resolution of the endoscopy image is thus reduced forthe method steps (and ultimately for the determination of the evaluationmeasure); for example a 3×3 field of pixels in the endoscopy image isrespectively averaged or, respectively, defined as a single image point.

In a preferred embodiment of the method, the evaluation measure isdetermined for multiple endoscopy images. Within the scope of anendoscopy, not just a single endoscopy image but rather an image seriesis normally created. The evaluation measure can then be determined foreach image (or at least multiple images) of the image series. Forexample, image points can hereby be disregarded that were alreadydepicted in a prior endoscopy image.

In a preferred embodiment, the method can also be applied to anendoscopy image which, as a sum image, is composed from individualimages. For example, the image series acquired in the course of anendoscopy can be assembled (within the scope of a “stitching”) into asingle image (enlarged relative to the individual images) of the entireendoscoped patient region, and the method can be applied to theassembled image.

It is then taken into account whether specific segments of the patientare depicted twice in multiple endoscopy images. The double or multipledepiction of the same points of the patient surface can correspondinglyaffect the determination of the color values (averaging over variouspixels, i.e. various viewing angles of one and the same point of thepatient) or can also affect the calculation of the patient area for thearea proportion or the total area, for example.

In a further preferred embodiment, a 3D data set is used as an endoscopyimage. The image can be designed as a virtual 3D image surface,particularly given assembly or, respectively, stitching of multipleendoscopy images of a three-dimensional hollow organ. The method canalso be applied to such a data set.

Naturally, a calibration of the method normally takes place (for exampleusing a RAL color map) in order to respectively obtain results that arecomparable between different patients or different endoscopy images.

BRIEF DESCRIPTION OF THE DRAWING

The single figure schematically illustrates implementation of anendoscopy procedure in accordance with the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The figure shows a section from a patient 2, namely a segment from hisesophagus 4. An endoscopy is presently conducted on the patient 2, whichis why an endoscope 6 is introduced into the esophagus 4. A camera 10that delivers an endoscopy image 12 of the inside of the patient ismounted at the end 8 of the endoscope. The viewing angle of the camera10 as well as the region 14 of the inner wall of the esophagus 4 thatare depicted in the endoscopy image 12 are represented with dashed linesin FIG. 1. The endoscopy image 12 is made up of individual pixels, ofwhich the pixels 16 a-d are shown as examples.

The endoscopy image 12 is now processed according to the methodaccording to the invention: for each pixel 16 a-d a check is initiallymade as to whether it depicts a body region of the patient 2. In theexample this is satisfied for all pixels. Each pixel 16 a-d thereforerepresents an image point 18 a-d that is to be examined further. Anassociated color value 20 a-d is now determined for each image point 18a-d. For each pixel 16 a-d, its respective color information isevaluated for this.

In a first embodiment, the color information of each pixel 16 a-d isrespectively a triplet of values, namely an RGB value 22 whichrespectively reflects the red, green and blue proportion of the color ofthe pixel 16 a-d. The image points 18 a and c respectively have thevalue A as an RGB value 22, the image point 18 b has the value B and theimage point 18 d has the value C. These yield the values of the colorvalues 20 a-d. The value A hereby corresponds to a dark red, the value Bcorresponds to a medium red and the value C corresponds to a light red.

An area proportion 24 a-d is now determined for each image point 18 a-d.In a first exemplary embodiment, this area proportion 24 a-d isrespectively oriented towards the area of 0.01 mm² of the respectivepixels 16 a-c in the endoscopy image 12. A pixel is associated with arelative area proportion of “1”.

This process is now repeated for all 320×200 pixels of the endoscopyimage 12. In the example another fourth value D is hereby found as acolor value (a dark grey).

For each found value A-D of the color values 20 a-d, the associatedcolor area 26A-D of the respective image points 18 a-d having thecorresponding value A-D is now determined via summation. In the examplewith pixels 16 a-d of equally large area, a color area 26A thus resultsfor the value A of “2” that is twice as large as the respective colorareas 26B and C with “1”. This process is now also implemented for all320×200 pixels of the endoscopy image 12.

In a last step, an evaluation measure 28 is determined from the colorareas 26A-D. In a first embodiment, this represents a histogram 30 ofthe respective color areas 26A-D over the color values A-D (for theentire image now, for example).

In an alternative embodiment, the entire area 32 of the endoscopy image12 is additionally computed. Given 320×200 pixels, this yields a valueof “64000”. As an alternative evaluation measure 28, here the respectiveratios 34 of the respective color areas 26A-D to the area 32 arecalculated.

The figure also shows the region 14 in the esophagus 4 of the patient 2which is depicted in the endoscopy image 12 as well as imaging regions36 a-d of the patient 2 which are depicted in the respective pixels 16a-d. In an alternative embodiment, the area proportions 24 a-d aredetermined correlated with the areas of the imaging regions 36 a-d andthe area 32 is determined correlated with the area of the region 14. Thecorrespondingly determined areas or, respectively, area proportion thenbetter reflect(s) the real area of the patient that is imaged, and notonly the image area in the endoscopy image 12. For example, an areaproportion 24 a of “1.1” results here since the imaging region 36 a liesat an angle relative to the viewing angle of the camera 10. The value“1” results for the area proportion 24 c since this area portion of thepatient 2 is aligned orthogonal to the camera 10. Due to the non-planaralignment relative to the camera 10, an area measurement of “66000”results for the total area 32.

In an alternative embodiment, the respective color values 20 a-d are notdetermined directly from the values of the pixels 16 a-d correspondingto the RGB values 22; rather, they are determined using a mask 38. TheRGB values 22 are mapped by the mask 38 to two classes of color values Aand B, wherein each color value hereby corresponds to a pathologicalproperty of the patient 2. The value A thus means that the correspondingclassified pixel is such a pixel that depicts an inflamed region of thepatient. A value B corresponds to a region of the patient that is notinflamed.

Therefore, not only individual RGB values 22 but respective value rangesin the RGB values 22 are also mapped to the values A and B using themask 38. A standard color system 40 (for example the RAL color system)is used for this.

The figure schematically shows an additional embodiment in whichrespective groups of pixels in the endoscopy image 12 are assembled as agroup 42, and a respective group 42 is depicted in an image point 18 a-dand this is processed according to the cited method.

The figure shows with dashed lines a situation at a later point in timeat which the endoscope 6 has been displaced further into the esophagus4. Here additional regions 14 of the patient 2 are depicted inadditional endoscopy images 12. The method described above can then beimplemented at the individual endoscopy images 12 in order to generatean evaluation measure 28 for each endoscopy image 12.

Alternatively, the endoscopy images 12 can also be assembled into a sumimage 44, or a 3D data set 46 can be reconstructed from these. Themethod described above can then be implemented on the sum image 44 or onthe 3D data set 46, and an evaluation measure 28 can respectively begenerated with regard to this.

Although modifications and changes may be suggested by those skilled inthe art, it is the intention of the inventors to embody within thepatent warranted heron all changes and modifications as reasonably andproperly come within the scope of their contribution to the art.

We claim as our invention: 1-17. (canceled)
 18. A method to process adigital endoscopy image of a patient comprising: in a processor suppliedwith an endoscopy image comprised of a plurality of pixels,automatically identifying pixels, among said plurality of pixels, asimage points that depict a body region of a patient; in said processordetermining a color value for each image point using a value of thepixel associated therewith; in said processor, determining an areaportion for each image point; in said processor, determining, for eachcolor value, a color area as a sum of all area portions of respectiveimage points having the respective color value; and in said processor,implementing an evaluation measure of said endoscopy image using therespective color areas.
 19. A method as claimed in claim 18 comprisingdetermining said area portion as a value correlated with an area of thepixel associated with the respective image point.
 20. A method asclaimed in claim 18 comprising determining said area portion as a valuecorrelated with an area of the patient depicted by a respective imagepoint.
 21. A method as claimed in claim 18 comprising implementing saidevaluation measure by determining a total area as a reference area forthe respective color areas.
 22. A method as claimed in claim 21comprising determining said total area as an entirety of an image areaof said endoscopy image.
 23. A method as claimed in claim 21 comprisingdetermining, as said total area, an area of the patient depicted in saidendoscopy image.
 24. A method as claimed in claim 21 comprisingimplementing said evaluation measure by calculating a quotient of arespect color area and said total area, for at least one of said colorareas.
 25. A method as claimed in claim 18 comprising implementing saidevaluation measure by determining a histogram of said color areas overrespective values of the color areas.
 26. A method as claimed in claim18 comprising selecting said color value to be correlated with apathological property of the patient.
 27. A method as claimed in claim18 comprises employing value ranges of respective pixel values as saidcolor values.
 28. A method as claimed in claim 27 comprising using astandard color system to determine said value ranges.
 29. A method asclaimed in claim 18 comprising determining the respective values of saidpixels as values of a plurality of color channels determined using amask applied to said color channels.
 30. A method as claimed in claim 18comprising selecting the respective color values as respectiveluminosity-normalized 2D values.
 31. A method as claimed in claim 18comprising determining said image points as groups of pixels in saidendoscopy image.
 32. A method as claimed in claim 18 comprisingimplementing said evaluation measure for a plurality of differentendoscopy images supplied to said processor.
 33. A method as claimed inclaim 18 comprising assembling said endoscopy image as a sum image of aplurality of individual endoscopy images.
 34. A method as claimed inclaim 18 comprising employing a 3D data set as said endoscopy image.